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Demonstration of a High-Fidelity Predictive/Preview 
Display Technique for Telerobotic Servicing in Space 

Won S. Kim and Antal K. Bejczy 

Abstract — A highly effective predictive/preview display technique for 
telerobotic servicing in space under several seconds communication time 
delay has been demonstrated on a large laboratory scale in May 1993, 
involving the Jet Propulsion Laboratory as the simulated ground control 
station and, 2500 miles away, the Goddard Space Flight Center as the 
simulated satellite servicing set-up. The technique is based on a high- 
fidelity calibration procedure that enables a high-fidelity overlay of 3-D 
graphics robot arm and object models over given 2-D TV camera images 
of robot arm and objects. To generate robot arm motions, the operator 
can confidently interact in real time with the graphics models of the robot 
arm and objects overlaid on an actual camera view of the remote work 
site. The technique also enables the operator to generate high-fidelity 
synthetic TV camera views showing motion events that are hidden in a 
given TV camera view or for which no TV camera views are available. 
The positioning accuracy achieved by this technique for a zoomed-in 
camera setting was about ±5 mm, well within the allowable ±12 mm 
error margin at the insertion of a 45 cm long tool in the servicing task. 

I. Introduction 

There is a genuinely growing interest to reduce extravehicular 
activity (EVA) astronauts time for servicing operations in low Earth 
orbit. One way to accomplish this is to introduce ground-controlled 
servicing by using telerobotic devices and techniques. In such op- 
erations, however, there is an unavoidable two-way communication 
time delay between a ground control station and the telerobotic device 
working in low Earth orbit. When the existing NASA telecommunica- 
tion facilities are utilized, this round-trip communication time delay 
can be expected to vary between four to eight seconds. This size 
of time delay prevents any effective closed-loop control interaction 
between the ground operator and the telerobotic device. 

Due to the very nature of satellite or other space servicing in 
low Earth orbit, “blind” (that is, fully pre-programmed and visually 
not supervised) automation is not possible for such tasks, and fully 
autonomous performance of such tasks is still far down the road. 
There is a need, therefore, to consider and apply an operator- 
interactive remote control method enabling an efficient crossing of 
the communication time delay barrier and still preserving operation 
reliability and safety. 

Typical teleoperation tasks require the operator’s visual and kines- 
thetic presence (real-time visual and force-feeling attention) to guide 
and control the remote robot arm’s actions. Control methods and 
schemes have been proposed and analyzed both theoretically and 
experimentally in the past twenty or so years to compensate in 
real time for the operator’s visual and kinesthetic absence in time- 
delayed teleoperation. (The different methods are reviewed elsewhere 
in this special issue; cf. [5]). There are two important and promising 
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schemes for enhancing telemanipulation capabilities under communi- 
cation time delay conditions: predictive/preview displays and shared 
compliance control. For details, see above quoted reference and also 
[1], [2], and [3]. 

In this paper, we briefly present the demonstration and evaluation 
of a space telerobotic servicing task performed under communication 
time delay on a large laboratory scale in May, 1993. In the demonstra- 
tion, a high-fidelity predictive/preview display technique developed 
at the Jet Propulsion Laboratory (JPL) was used combined with wrist 
force sensor referenced compliance control implemented at the other 
NASA center (Goddard Space Flight Center, GSFC) participating in 
the demonstration experiments. In the demonstration, JPL acted as the 
operator site simulating the ground control station, and GSFC, more 
than 2500 miles away from JPL, acted as the remote work site with 
a life-size satellite servicing task mock-up. During the demonstration 
experiments, TV camera images from the GSFC work site were sent 
back to the JPL control station over the NASA Select Satellite TV 
channel at live video rate (30 frames/s). Command and control data 
from the JPL control station to the GSFC work site and execution 
status and sensor data from GSFC to JPL were sent as “data blocks” 
through the Internet computer communication network. 

First, we briefly describe the task and the task set-up at GSFC, 
followed by a brief summary of the salient features of the high-fidelity 
predictive/preview display technique developed at JPL and used in 
the joint JPL-GSFC demonstration experiments. Then, we briefly 
summarize the results. The conclusion will also indicate planned 
future work. 

II. Remote Servicing Task Set-Up 

The servicing demonstration task to be performed at GSFC, 
remotely controlled from JPL, was the exchange of a nearly four feet 
tall, four feet wide and two feet deep exchangeable module (called 
Orbital Replacement Unit, ORU) on a satellite mock-up. This module 
exchange task was originally designed to be performed by astronauts 
working in pressurized space suits in the Space Shuttle cargo bay. 
A robot arm (Robotics Research Corporation K-1607) was placed in 
front of the exchange module of the satellite’s life-size set-up. The 
arm was equipped with an eighteen inch (about 45 cm) long power 
screwdriver (called lightweight servicing tool, LST), which also could 
be rigidly connected to the exchange module to remove the module 
from the satellite. The basic robot control task was to insert the power 
screwdriver through an eighteen inch long hole to reach the module’s 
latching mechanism at the module’s back plane, then to unlatch the 
module from the satellite’s frame, and rigidly connect the module to 
the robot arm in order to remove the module from the satellite. The 
placement of a new module back to the satellite’s frame followed 
the reverse sequence of actions. The RRC K-1607 arm of GSFC was 
equipped with a 6-dof wrist force sensor. The LST and the “new” 
ORU module were equipped with GSFC’s capaciflector proximity 
sensor system. 

The remote control from JPL utilized three fixed TV camera 
settings at the GSFC test site: the first one was located at the 
base of the robot arm looking straight ahead towards the exchange 
module in the satellite. The second one was located at the right and 
perpendicular to the first camera view at nearly the same elevation. 
The third one was placed halfway in between and about 35 degrees 
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above the previous two camera settings to provide an oblique view of 
the work scene. Two zoom settings were used for the third camera. 

The ORU changeout task scenario used in the remote servicing 
demonstration had the following sequence: 

1) Move the arm from the starting position to a position where 
the LST tip is about 20 cm in front of the entrance of the hole 
on the ORU module. 

2) Move the LST to the immediate entrance of the hole. 

3) Insert the LST. 

4) Latch the LST to the ORU. 

5) Turn on the power tool to loosen the screw. 

6) Pull out the ORU by 5 cm. 

7) Continue to withdraw the ORU so that it is about 15 cm apart 
from the satellite. 

8) Move the ORU to a stow position. 

9) Move the ORU back to 15 cm in front of the satellite frame. 

10) Align the ORU for insertion. 

11) Insert the ORU. 

12) Turn on power tool to tighten the screw. 

13) Unlatch the LST from the ORU. 

14) Pull out the LST to about 20 cm away from the ORU. 

15) Finally, move the arm back to the starting position. 

To compensate in real time for the operator’s time-delayed visual 
perception of the task, a high-fidelity predictive/preview display 
technique was used. This technique provides a high-fidelity overlay of 
computer graphics represented virtual reality over TV camera views 
represented actual reality. The operator generated the overlaid robot 
graphics image motions by a hand controller or by computer control 
algorithms. Then the operator visually verified the correctness of 
the generated robot motions through previewing the simulated robot 
graphics image motions embedded into the monitor of an actual 
TV camera image of the work scene. Once verified, the recorded 
motion command is sent over to the GSFC robot control system 
for execution there. The robot motion execution during the actual 
contact phase of tool or module insertion or removal actions is aided 
by automatic compliance control referenced to wrist force sensor 
data, implemented at GSFC. This compensates in real-time for the 
operator’s kinesthetic absence. Capaciflector proximity sensor data 
can play the role of automatically preventing unforeseen or accidental 
contact errors during near-contact but still in free-space robot motion. 

III. High-Fidelity Matching of Virtual 
Reality to the Actual One 

The process of matching the virtual reality, represented by com- 
puter graphics images, to the actual reality, as shown on TV camera 
images, is composed of four major high-fidelity technical components 
in the time-delayed telemanipulation method developed at JPL. First, 
the method requires the construction of high-fidelity 3-D graphics 
models of both the robot arm and the objects of interest for robot 
manipulator tasks. Second, which is the cornerstone of the method, 
the 3-D graphics models are calibrated with high-fidelity to the given 
2-D TV camera views that cover the sight of both the robot arm 
and the objects of interest. Third, which is a basic requirement of the 
method, the calibrated graphics models are overlaid with high fidelity 
over the actual robot arm and object images as shown to the operator 
on the TV camera monitor screen. Fourth, the method requires control 
of the motion of the robot arm graphics image on the screen by the 
same control software that controls the motion of the real robot, in 
order to assure high fidelity in the formulation of motion commands. 

This time-delayed telemanipulation method was designed to pro- 
vide a high visual confidence to the operator when the operator 
interacts with graphics images in real time in generating, that is, in 


predicting, or in previewing manually or computer generated robot 
arm motion before the actual robot arm motion takes place. This 
method goes beyond the general practice which uses graphics models 
displayed separately from actual TV views. Our calibrated overlay 
method in essence provides visual verification and updates for the 
validity of graphics models for an actual live scene. The calibrated 
virtual reality display system can also provide synthetic TV camera 
views to the operator. Synthetic TV camera views can make critical 
motion events or robot arm position and alignment states visible 
which otherwise would be hidden from the operator in a given TV 
camera view or for which no real TV camera views are available. 
The calibrated virtual reality method also enables the placement of 
calibrated 3-D target symbols over the live TV video helping the 
real-time visual guidance/control and real-time visual verification by 
the operator. 

Construction of high-fidelity graphics models is possible through 
state-of-the-art commercial graphics software systems. However, no 
commercial graphics software system is available which is easily 
compatible with the calibration and overlay requirements in our time- 
delayed telemanipulation method. Therefore, a graphics modeling 
software system (called HYDRA) was developed in-house in the past 
few years to have an integrated modeling, calibration, and overlay 
working system. 

A. Camera Calibration 

Accurate matching of graphics models to the actual task environ- 
ment based on TV camera views requires camera calibration and 
object localization. Although there are several camera calibration and 
object localization techniques, none of them specifically address the 
practical problem of calibration of graphics models for overlay on 
live video for predictive/preview display applications. Therefore, a 
reliable operator-interactive camera calibration procedure was devel- 
oped [4]. In this procedure, the robot arm itself is used as a calibration 
fixture. The operator enters the correspondence information between 
3-D graphics model points and 2-D camera image points of the arm to 
the computer by repeatedly clicking with a mouse a graphics model 
point and then its corresponding image point for each corresponding 
pair. (That is, the current calibration is based on a point-to-point 
mapping technique; the selected points are visually sharply defined 
characteristic points.) To improve the calibration accuracy over 
the normal operating space of the arm, the operator enters the 
corresponding data points for several different poses of the arm within 
the same camera view. Thereafter, the system computes the camera 
calibration parameters. This calibration procedure is repeated for each 
of the camera views that are needed for a given task. 

The computation of the camera calibration parameters is based on 
the ideal pinhole model of image formation by the camera. Under 
this assumption, the image format of the camera is described by a 
linear perspective projection, and the relation between 3-D graphics 
model points and the corresponding 2-D TV camera image points can 
be described by a single linear 4x3 calibration matrix. In general, 
it would be physically more accurate to consider the second-order 
nonlinear radial distortion of the camera lens optics in the model of 
camera image formation for calibration purposes. However, this more 
elaborate model leads to a nonlinear projection which, in general, is 
not supported by real-time 3-D graphics workstations. For practical 
reasons, therefore, the ideal pinhole model was used in the camera 
calibration procedure for overlaid preview/predictive displays. 

In our implementation, a Silicon Graphics workstation (IRIS 
4D/310 VGX) with a square-pixel resolution monitor and with a 
VideoLab Board are used for the video image capture and for the 
calibrated graphics overlay on the live video image. In computing 
camera parameters using the above hardware, we further assumed 
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that i) the camera optical axis is perpendicular to the image plane, 
passing through the center of the camera view, and ii) the viewpoint 
of the graphics overlay window coincides with the full size of the 
camera view window. We note here that the final camera calibration 
matrix defines the position, orientation, and field of view angle of 
the camera. 

The actual camera calibration computations in our method can be 
carried out by a combination of linear and nonlinear least-squares 
algorithms. The linear algorithm in general does not guarantee the 
orthonormality of the rotation matrix, providing only an approximate 
solution. The nonlinear algorithm provides the least-squares solution 
that satisfies the orthonormality of the rotation matrix, but requires 
a good initial guess to converge to the desired least-squares solution 
without a very time-consuming random search. When an approximate 
solution is known, one can start with the nonlinear iterative algorithm 
directly by using the approximate solution as an initial guess. When 
an approximate solution is unknown, however, the linear algorithm 
is used to find first an approximate solution, and then the nonlinear 
algorithm is applied. 

B. Object Localization 

In our method of high-fidelity matching of virtual reality to 
the actual one, the operator performs object localization after the 
calibration of all desired camera views. For determining the object 
3-D pose from a given 2-D camera view, several algorithms are 
available in the literature. In our procedure, a projection-based 
linear/nonlinear algorithm is extended to allow object localization 
for any number of multiple camera views. 

The operator’s interactive data entry procedure to feed inputs to 
the extended object localization algorithms is essentially identical 
to the one for camera calibration, except that the operator enters 
corresponding data points for an object (not the robot arm this time) 
with several different camera views. Again, a combination of linear 
and nonlinear algorithms is used for computing the object pose 
(position and orientation), and the computational procedure depends 
upon whether an approximate initial solution is known. More details 
on the camera calibration and object localization applied in our 
high-fidelity predictive/preview display technique can be found in [4]. 

Once the camera calibration and object localization are completed, 
the graphics models of both the robot arm and the object can be 
overlaid with high fidelity on the corresponding actual video images 
in a given video camera view. The arm and object graphics models 
can be overlaid in a wire-frame or in a solid-shaded polygonal 
rendering, with varying levels of transparency, providing different 
visual effects to the operator for different task details. The hidden 
lines can be removed or retained by the operator, dependent on the 
information needs in a given task. In many cases, retained hidden 
lines can provide very useful and needed information to the operator. 

C. Motion Control of Robot Arm’s Graphics Image 

The idea with the high-fidelity graphics overlay image is that the 
operator can interact with it visually in real time when manually gen- 
erating motion commands or when previewing computer generated 
trajectories, and doing it against the live video scene on a monitor 
within one perceptive frame. Thus, this method would compensate in 
real-time for the operator’s time-delayed visual perception. Typically, 
the geometric dimensions of a monitor and geometric dimensions of 
the real work scene shown on the monitor are quite different. For 
instance, an 8-inch long trajectory on a monitor can correspond to a 
24-inch long trajectory in the actual work space, that is, three times 
longer than the apparent trajectory on the monitor screen. Therefore, 
in order to preserve fidelity between previewed graphics arm image 



Fig. 1 . Graphics user interface during robot arm graphics image calibration. 

and actual arm motions, all previewed actions on the monitor were 
scaled down very closely to the expected real motion rate of the arm 
hardware. The manually generated trajectories were also previewed 
before sending the motion commands to the GSFC control system in 
order to verify that all motion data were properly recorded. Preview 
displays contribute to operational safety. In order to eliminate the 
problem associated with the varying time delay in data transfer, the 
robot motion trajectory command is not executed at the GSFC control 
system until all the data blocks for the trajectory are received. 

An element of fidelity between graphics arm image and actual arm 
motion was given by the requirement that the motion of the graphics 
image of the arm on the monitor screen be controlled by the same 
software that controls the motion of the actual arm hardware. This 
required the implementation of the GSFC control software in the JPL 
graphics computer. 

A few seconds after the motion commands were transmitted to 
GSFC from JPL, the JPL operator could view the motion of the 
real arm on the same screen where the graphics arm image motion 
was previewed. If everything went well, the image of the real 
arm followed the same trajectory on the screen that the previewed 
graphics arm image motion previously described, and the real arm 
image motion on the screen stopped at the same position where 
the graphics arm image motion stopped earlier. After completion of 
robot arm motion, the graphics images on the screen were updated 
with the actual final robot joint angle values. This update eliminates 
accumulation of motion execution errors from the graphics image 
of robot arm, and retains graphics robot arm position fidelity on 
the screen even after the completion of a force sensor referenced 
compliance control action. 

The demonstration tasjc involved was to connect a large payload to 
the robot arm, move it away from the satellite frame and then place a 
new payload (the same payload in our demo) back to the original posi- 
tion of the previous payload. This required connecting/disconnecting 
robot arm and payload graphics images, and to graphically simulate 
motion of payload held by the robot arm. If the robot arm deflection 
caused by the weight of a payload is not modeled, that can decrease 
the graphics fidelity of the payload graphics model trajectory and the 
graphics fidelity of the final position of the payload graphics model 
on the screen. This can cause problems when the payload has to 
be inserted under tight geometric tolerances. We encountered this 
problem during the demonstration experiments and compensated for 
it by a trial and error approach. 
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Fig. 2. A calibrated robot arm overlay in wire-frame rendering. 



Fig. 3. A synthetic TV view of the configuration shown in Fig. 2, after 
object localization. 


D. Operator Interface 

There is a wealth of computer related operator activities in a system 
that 1) intends to match virtual reality to the actual one with high 
fidelity and 2) is expected to remotely control a robot together with 
its live video/graphics system and with its selectable control modes. 
Therefore, a useful operational operator interface requires careful 
design considerations. We developed a graphical operator interface 
system using two Silicon Graphics workstations and one NTSC video 
monitor. The primary workstation (IRIS 4D/310 VGX) is used for 
preview/predictive displays and for various graphical user interfaces 
(GUI’s) in a four-quadrant format. The second workstation (IRIS- 
4D/70 GT) is solely used for sensor data display. More on the operator 
interface can be found in [41. 

Fig. 1 shows the four-quadrant GUI during the arm calibration 
process. Fig. 2 shows the corresponding calibrated overlay result. 
Fig. 3 shows a synthetic TV view of the same configuration that is 
shown in Fig. 2, after object localization. Note that this synthetic TV 
view tells much more than Fig. 2 about the relative distance between 
tool tip and hole. Fig. 4 shows a display situation for previewing the 
end point of a generated motion for tool insertion. Fig. 5 shows the 


Fig. 4. Predictive/preview display of end point of first motion task including 
object overlay in wire-frame rendering. 



Fig. 5. Same configuration as in Fig. 4, but after motion execution and 
from a different TV camera view. 


same preinsertion status from a different camera view, while Fig. 6 
shows it in a zoomed-in mode. 

IV. Results 

Four camera views were calibrated for the May 1993 JPL-Goddard 
remote servicing demonstration. In each camera calibration, the 
operator typically entered about 15 to 30 data points in total from 
3 or 4 different arm poses. For both the side-view and oblique- view 
cameras, the y-axis field-of-view angles were approximately fovy = 
32°, and the average calibration errors between the projections of 3-D 
object points on the image plane and the actual 2-D image points were 
typically in the range of 0.5 0.7% (with 1. 6-2.0% maximum errors). 
The object (robot arm) distance from these cameras was about 3 m, 
and the 0.6% average error on the image plane corresponds to 1 
cm displacement error on the hypothetical plane 3 m in front of the 
camera. Two zoom settings were used for the overhead (front-view) 
camera, which was about 1 m away from the robot end effector. For 
the wide-angle view (fovy = 38°), the average error on the image 
plane was typically 0.7-0.9% (2.0-2.8% maximum error), and the 
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Fig. 6. Same configuration and TV camera view as in Fig. 5, but camera 
zoomed in. 


0.8% average error corresponds to 0.5 cm displacement error on the 
plane 1 m in front of the camera. For the zoom-in view (fovy = 8°), 
the average error on the image plane was typically 1.2-1. 6% (3.2- 
4.2% maximum error), and the 1.4% average error corresponds to 
0.2 cm displacement error on the plane 1 m in front of the camera 
(See Fig. 6 for this setting). 

The average object localization errors on the image planes were 
about 0.9-1. 9% (3. 0-7.0% maximum errors). The 1.4% average 
error on the image planes corresponds to 0.2 cm displacement error 
with respect to the zoom-in overhead camera view, and to 2.5 cm 
displacement error with respect to the side-view or oblique-view 
cameras, in wide field-of-view settings. 


V. Conclusion 

The demonstration experiments have been performed successfully, 
showing the practical utility of high-fidelity predictive/preview dis- 
play techniques, combined with compliance control, for the type of 
telerobotic servicing tasks in space that were shown in the May, 1993 
demonstration. The same techniques also have a wide range of terres- 
trial application possibilities. Future work will include: 1) simulated 
tests on other space application tasks like Hubble Space Telescope 
Servicing, and 2) interactive model building and intermittent model 
matching updates using model-based image processing. 
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